1 Definition

We have discussed statistical models. Now we want to study a family of models with a specific structure, and they have a lot of properties.

By definition, take integral on both sides w.r.t ${\displaystyle x}$ over ${\displaystyle \mathcal{X}}$: $$\begin{array}{rl}& 1={\int }_{\mathcal{X}}{\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)-A(\eta )}h(x)\mathrm{d}\mu (x)\\ \text{(1.2)}& \Rightarrow & A(\eta )=\log ({\int }_{\mathcal{X}}{\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)}h(x)\mathrm{d}\mu (x))\le \mathrm{\infty }.\end{array}$$
We should restrict ${\displaystyle A(\eta )<\mathrm{\infty }}$, so define natural parameter space $${\mathrm{\Xi }}_1=\{\eta |A(\eta )<\mathrm{\infty }\}\subset {\mathbb{R}}^s.$$
We can prove that ${\displaystyle A(\eta )}$ is convex function, so ${\displaystyle {\mathrm{\Xi }}_1}$ is convex set.

1.1 Distribution of ${\displaystyle {\displaystyle T(X)}}$

If ${\displaystyle X\sim p_{\eta }(x)={\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)-A(\eta )}}$ w.r.t ${\displaystyle \mu }$ (WLOG we let ${\displaystyle h\equiv 1}$, otherwise we let ${\displaystyle \mu }$ to absorb ${\displaystyle h}$), then ${\displaystyle T(X)\sim q_{\eta }(t)={\mathrm{e}}^{{\eta }^{\mathrm{T}}t-A(\eta )}}$ w.r.t ${\displaystyle \nu }$, where ${\displaystyle \nu }$ is the measure ${\displaystyle \mu }$ push forward through ${\displaystyle T:\mathcal{X}\to {\mathbb{R}}^s}$: $$\nu (B)\stackrel{\mathrm{\Delta }}{=}\mu (\{x:T(x)\in B\}).$$ So $$\begin{array}{rl}{\mathbb{P}}_{\eta }(T(X)\in B)& =\int {\mathbf{1}}_B(T(X)){\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)-A(\eta )}\mathrm{d}\mu (x)\\ & =\int {\mathbf{1}}_B(t){\mathrm{e}}^{{\eta }^{\mathrm{T}}t-A(\eta )}\mathrm{d}\nu (t).\end{array}$$
This is simplest in discrete case (we can now drop ${\displaystyle h\equiv 1}$ assumption):

and we denote $$\nu (\{t\})=\sum _{x:T(x)=t}h(x)\mu (\{x\}).$$

1.2 Carnonical Form

Based on discussion above, we can simplify the structure of exponential family:

• ${\displaystyle T(x)\to x}$ (by sufficiency reduction discussed above)
• ${\displaystyle h(x)\to 1}$ (by absorbing ${\displaystyle h}$ into ${\displaystyle \mu }$)
• ${\displaystyle \theta =\eta }$ (by parameterizing by ${\displaystyle \eta }$)

Based on these, we define

2 Differential Identites

By (1.2) we have $$\begin{array}{}\text{(2.1)}& {\mathrm{e}}^{A(\eta )}={\int }_{\mathcal{X}}{\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)}h(x)\mathrm{d}\mu (x).\end{array}$$ We can differentiate this function to get meaningful results. We use without proof that it's correct to swap differentiation and integral within ${\displaystyle {\mathrm{\Xi }}_1}$.

2.1 Mean of ${\displaystyle {\displaystyle T(X)}}$

Denote ${\displaystyle T_j(x)}$ be the ${\displaystyle j}$ th coordinate of ${\displaystyle T(x)}$. Then $$\begin{array}{rl}\frac{\partial }{\partial {\eta }_j}{\mathrm{e}}^{A(\eta )}& ={\int }_{\mathcal{X}}\frac{\partial }{\partial {\eta }_j}{\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)}h(x)\mathrm{d}\mu (x)\\ \Rightarrow {\mathrm{e}}^{A(\eta )}\frac{\partial A}{\partial {\eta }_j}(\eta )& ={\int }_{\mathcal{X}}{T}_j(x){\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)}h(x)\mathrm{d}\mu (x)\\ \Rightarrow \frac{\partial A}{\partial {\eta }_j}(\eta )& ={\int }_{\mathcal{X}}{T}_j(x){\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)-A(\eta )}h(x)\mathrm{d}\mu (x)={\mathbb{E}}_{\eta }[T_j(X)].\end{array}$$Rearrange for ${\displaystyle j=1,\cdots ,s}$: $$\begin{array}{}\text{(2.2)}& \mathrm{\nabla }A(\eta )={\mathbb{E}}_{\eta }[T(X)].\end{array}$$

2.2 Variance of ${\displaystyle {\displaystyle T(X)}}$

Take a second partial derivative: $$\begin{array}{rl}\frac{{\partial }^2}{\partial {\eta }_j\partial {\eta }_k}{\mathrm{e}}^{A(\eta )}& ={\int }_{\mathcal{X}}\frac{{\partial }^2}{\partial {\eta }_j\partial {\eta }_k}{\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)}h(x)\mathrm{d}\mu (x)\\ \Rightarrow {\mathrm{e}}^{A(\eta )}(\frac{{\partial }^{2}A}{\partial {\eta }_j\partial {\eta }_k}+\frac{\partial A}{\partial {\eta }_j}\frac{\partial A}{\partial {\eta }_k})& ={\int }_{\mathcal{X}}{T}_j(x)T_k(x){\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)}h(x)\mathrm{d}\mu (x)\\ \Rightarrow \frac{{\partial }^{2}A}{\partial {\eta }_j\partial {\eta }_k}+{\mathbb{E}}_{\eta }[T_j(X)]{\mathbb{E}}_{\eta }[T_k(X)]& ={\mathbb{E}}_{\eta }[T_j(X)T_k(X)]\\ \Rightarrow \frac{{\partial }^{2}A}{\partial {\eta }_j\partial {\eta }_k}& ={\mathrm{Cov}}_{\eta }(T_j(X),T_k(X)).\end{array}$$ Finally we get $$\begin{array}{}\text{(2.3)}& {\mathrm{\nabla }}^{2}A(\eta )={\mathrm{Var}}_{\eta }(T(X)).\end{array}$$ Here ${\displaystyle {\mathrm{Var}}_{\eta }(T(X))}$ is a ${\displaystyle s\times s}$ covariance matrix of the random vector ${\displaystyle T(X)}$.

2.3 MGF of ${\displaystyle {\displaystyle T(X)}}$

Moment Generating Function (MGF) of a ${\displaystyle d}$ dimensional random vector ${\displaystyle X\sim P}$ is defined as ${\displaystyle M_X(u)=\mathbb{E}[{\mathrm{e}}^{u^{\mathrm{T}}X}],u\in {\mathbb{R}}^d}$. Note that 1-dim case is introduced in here. We can calculate moments by taking derivative of MGF, as long as ${\displaystyle M_X(u)}$ is well-defined for a neighborhood of ${\displaystyle 0}$. Now we evaluate the first moments $$\begin{array}{r}\frac{\partial }{\partial u_j}{M}_X(u)={\int }_{\mathcal{X}}\frac{\partial }{\partial u_j}{\mathrm{e}}^{u^{\mathrm{T}}x}\mathrm{d}P(x)={\int }_{\mathcal{X}}{x}_j{\mathrm{e}}^{u^{\mathrm{T}}x}\mathrm{d}P(x).\end{array}$$
Let ${\displaystyle u=0}$, we obtain $$\frac{\partial }{\partial u_j}{M}_X(0)={\int }_{\mathcal{X}}{x}_j\mathrm{d}P(x)=\mathbb{E}[X_j].$$ Similarly $$\begin{array}{rl}& {\frac{{\partial }^{m_1+\cdots +m_d}}{\partial u_1^{m_1}\cdots \partial u_d^{m_d}}{M}_X(u)|}_{u=0}\\ \text{(2.4)}& =& {{\int }_{\mathcal{X}}{x}_1^{m_1}\cdots x_d^{m_d}{\mathrm{e}}^{u^{\mathrm{T}}x}\mathrm{d}P(x)|}_{u=0}=\mathbb{E}[X_1^{m_1}\cdots X_d^{m_d}].\end{array}$$
On the other hand, given ${\displaystyle \eta ,P_{\eta }}$, we can explicitly calculate the MGF for exponential family $$\begin{array}{rl}{M}_{T(X)}(u)& ={\mathbb{E}}_{\eta }[{\mathrm{e}}^{u^{\mathrm{T}}T(X)}]={\int }_{\mathcal{X}}{\mathrm{e}}^{u^{\mathrm{T}}T(x)-A(\eta )}h(x)\mathrm{d}\mu (x)\\ & ={\mathrm{e}}^{-A(\eta )}{\int }_{\mathcal{X}}{\mathrm{e}}^{(u+\eta {)}^{\mathrm{T}}T(x)}h(x)\mathrm{d}\mu (x)={\mathrm{e}}^{A(\eta +u)-A(\eta )}.\end{array}$$

2.4 CGF

The cumulant-generating function (CGF) is the log of MGF: ${\displaystyle K_X(u)=\log M_X(u)}$. So for exponential family, $$\begin{array}{rl}{K}_{T(X)}(u)& =A(\eta +u)-A(\eta )\\ \Rightarrow {\frac{\partial }{\partial {\eta }_j}{K}_T(u)|}_{u=0}& =0.\end{array}$$

3 Other Parameterizations

Instead of parameterizing ${\displaystyle \mathcal{P}}$ w.r.t ${\displaystyle \eta }$, we can parameterize the family by another ${\displaystyle \eta =\eta (\theta )}$, so $$p_{\theta }(x)={\mathrm{e}}^{\eta (\theta {)}^{\mathrm{T}}T(x)-B(\theta )}h(x),B(\theta )=A(\eta (\theta )).$$

4 Interpretation: Exponential Tilting

We can think of ${\displaystyle p_{\eta }(x)={\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)-A(\eta )}h(x)}$ as an exponential tilt for the carrier ${\displaystyle h(x)}$:

1. Start with carrier ${\displaystyle h(x)}$.
2. Multiply by ${\displaystyle {\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)}}$
3. Re-normalize by ${\displaystyle {\mathrm{e}}^{-A(\eta )}}$.

${\displaystyle T(X)=(T_1(X),\cdots ,T_s(X))}$ can be viewed as giving linear space of directions in which we can tilt ${\displaystyle h(x)}$. ${\displaystyle {\mathrm{\Xi }}_1}$ is all tilts after which normalization is possible (not going to infinity).

5 Repeated Sampling from Exponential Families

One of the most important properties of exponential families is that a large sample can be summarized by a low-dimensional statistic.
Suppose ${\displaystyle X=(X_1,\cdots ,X_n)}$ represents iid sample from an exponential family $$X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }{p}_{\eta }^{(1)}(x)={\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)-A(\eta )}h(x),$$ then $$p_{\eta }(x)=\prod _{i=1}^n{\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x_i)-A(\eta )}h(x_i)=\exp \{{\eta }^{\mathrm{T}}\sum _{i=1}^{n}T(x_i)-nA(\eta )\}\prod _{i=1}^{n}h(x_i).$$ This is an exponential family with sufficient statistic ${\displaystyle \sum _{i=1}^{n}T(X_i)}$, base density ${\displaystyle \prod _{i=1}^{n}h(x_i)}$ and log-partition function ${\displaystyle nA(\eta )}$.